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EXECUTIVE SUMMARY 

This report focuses on the initial design, implementation and baseline results of the five- 
year Longitudinal Educational Growth Study (LEGS) of the Milwaukee Parental Choice 
Program (MPCP) being conducted by the School Choice Demonstration Project (SCDP). The 
LEGS will be the first evaluation of the participant effects of the MPCP using student-level data 
to be implemented since the initial pilot program expanded dramatically in 1995. Included this 
initial report are baseline descriptions of achievement tests for a representative sample of MPCP 
students in grades 3 through 9, as well as outcomes for comparable samples of students in 
Milwaukee Public Schools (MPS). Also included are a descriptive analysis of survey results of 
MPCP parents and a carefully matched sample of MPS parents as well as a brief description of 
the results of student surveys for both samples. 

The first section of the baseline report discusses the construction of a sample of 2,727 
MPCP students in grades 3-9. The report also discusses the selection of 2,727 similar Milwaukee 
Public School students. For both samples the core of this longitudinal study will be to track the 
educational progress across the two samples through school year 2011-12. We demonstrate that 
the sample of MPS students constructed by the SCDP is more similar to the representative 
MPCP sample along demographic and initial achievement criteria than other potential 
comparison groups of MPS students. 

The baseline results indicate that MPCP students in grades 3 to 5 are currently scoring 
slightly lower on the math and reading portions of the Wisconsin Knowledge and Concepts 
Examinations (WKCE) than their MPS counterparts. However, no such difference exists for 
students in grades 6 to 8. Benchmark Test results for 9 th graders are also similar between the two 
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groups. The differences in grades 3 to 5 are almost exclusively due to lower MPCP math scores 
that disappear in grades 6 to 8. 

According to our surveys of parents and students, MPCP parents had lower incomes, but 
higher levels of education than MPS parents. The two groups were also quite similar on how 
they learned of their child's school and the qualities they sought in schools. A key difference 
was that MPCP parents got more information from churches and valued religious instruction 
more than MPS parents. In both groups, over 70 percent of students were attending their 
parents’ first choice of schools. Both MPCP and MPS parents and students showed high levels 
of satisfaction with their schools — in some cases higher than national averages. However, 

MPCP parents and students were generally more positive about their schooling experience than 
their counterparts in MPS. MPCP parents were less likely to report problems at school such as 
school violence, and had slightly higher educational expectations for their children, than 
comparable MPS parents. Students were also very positive about their schools, differing only 
slightly in their evaluation of their school climate depending on whether they were in the MPCP 
or MPS. 

The LEGS is the student-based component of the evaluation of the MPCP authorized in 
WI Act 125, Sec. 8, 1 19.23 (7)(e), 2 to be carried out by the SCDP based at the University of 
Arkansas. This project is being funded by a diverse set of philanthropies including the Annie E. 
Casey, Joyce, Kem Family, Lynde and Harry Bradley, Robertson, and Walton Family 
foundations. We thank them for their generous support and acknowledge that the actual content 
of this report is solely the responsibility of the authors and does not necessarily reflect any 
official positions of the various funding organizations, the University of Arkansas, the University 
of Wisconsin, or Westat, Inc. We also express our gratitude to MPS, the private schools in the 
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MPCP, and the state Department of Public Instruction for willing cooperation, advice, and 
assistance. 
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INTRODUCTION 

The Milwaukee Parental Choice Program 
The Milwaukee Parental Choice (Voucher) Program (MPCP) was the first parental 
choice program in the United States to allow urban parents to send their children to private 
schools with public funds covering the entire costs. 1 Often simply referred to as the “Choice” 
program, it is the largest of 12 voucher programs in the United States. In 2006-07, the baseline 
year of this report, a total of 17,749 students received vouchers that averaged $6,501 to attend 
one of 122 participating private schools in Milwaukee. 

The MPCP began in September 1990, based on legislation passed the year before as part 
of the biennial budget bill. For the first time in the United States, urban public school students 
were allowed to attend private schools with publicly funded vouchers. The program was 
targeted to low-income students (175% of the poverty line or less) who either had attended 
Milwaukee Public Schools (MPS) or were not in school the prior year. Only secular private 
schools were allowed to participate, and they had to limit their voucher students to 49 percent of 
total enrollment. The program was capped at 1 percent of the MPS student population 
(approximately 1,000 students in 1990). The voucher was equal to the annual per pupil MPS 



1 Since the late 1800s, Maine and Vermont have provided school vouchers to students in certain rural areas that 
lacked a public school, originally allowing them to attend any public or private school of their choosing in the area 
at public expense. The Maine program was subsequently limited to public or non-sectarian private schools. There 
is no clear consensus in the school choice literature regarding whether or not Maine and Vermont’s “town 
tuitioning” programs are actual voucher programs, though in concept and operation they are similar instruments of 
parental school choice. 

2 This count includes the town tuitioning programs in Maine and Vermont; means-test programs in Cleveland, the 
District of Columbia, and Milwaukee; disability programs in Arizona, Florida, Georgia, Ohio, and Utah; a foster- 
care program in Arizona; and a statewide program in Ohio limited to students attending public schools in need of 
improvement. Of these, only Florida’s McKay Scholarship Program for students with disabilities has an enrollment 
(16,812 in 2006-07) that rivals the MPCP. 
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state aid ($2,446 in 1990). Annual evaluations and reports were required and the program was to 
terminate with the 1994-95 year. 

There have been three major modifications in the program. In 1993-94 the maximum 
enrollment of Choice students in a school was raised from 49 to 65 percent and the total program 
cap was raised to 1.5 percent of the MPS enrollment. All sunset provisions were removed. 

Major changes in 1995 allowed sectarian schools to participate; allowed students who were in 
private schools in kindergarten through third grade to participate; eliminated any cap on voucher 
students per school; increased the program cap to 15 percent of MPS students; and eliminated all 
evaluation and reporting requirements. 

Another series of major changes was made in 2005, with Wisconsin Act 125. The 
students’ prior school criteria were completely dropped, so that any student living in Milwaukee 
was potentially eligible. The family income cap for continuing Choice students was raised to 
220 percent of the poverty line if they had been admitted earlier with an income under 175 
percent. The program limit was raised to 22,500 students. Private schools faced a set of new 
accreditation and testing requirements. Those new testing requirements of Act 125 were coupled 
with the call for an independent evaluation of the MPCP by the School Choice Demonstration 
Project, to be reviewed annually by the Legislative Audit Bureau. 3 

The general purposes of this report are two fold. The first is to outline the general 
research and methodological issues that will guide the research over the five-year period of the 
Longitudinal Educational Growth Study (LEGS). Those issues and the research design are 
focused on longitudinal changes in the educational outcomes of comparable students over the 
period. The second purpose is to present baseline descriptions of MPCP participants both in 

3 For more on the School Choice Demonstration Project’s study mandate, see Appendix A. 
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terms of baseline test scores, from which we will judge growth in achievement, and baseline 
survey results of samples of both Choice and MPS parents and students. 

Prior Research 

Prior research on publicly funded voucher programs in the United States has been limited 
to four locations: Milwaukee, Cleveland, Florida, and the District of Columbia. Privately 
funded voucher programs have also been studied, with most attention being given to programs in 
New York City, Washington, D.C. and Dayton, OH (Howell et al, 2002). Despite the limited 
number of voucher programs, numerous researchers have contributed a considerable body of 
evidence to the debate over vouchers. The type of research falls into two broad areas: 1) the 
differences in achievement and other outcomes between students in voucher schools and various 
comparison groups; and 2) the systemic, “competitive” effects of voucher programs on the 
remaining public schools. The LEGS deals only with the former question. Future reports from 
the School Choice Demonstration Project will address the latter area. 

The evidence on achievement from prior studies has yet to produce a strong academic 
consensus, with reported voucher impacts ranging from 0 to statistically significant gains of 3-4 
percentile points per year. Prior research on Milwaukee from 1990 to 1994 found mixed results 
on student achievement, with all researchers agreeing that there were not significant differences 
between private (voucher) students and a random sample of similarly eligible MPS students on 
reading scores even after as long as three years in the program. There was considerable 
disagreement on possible math differences, with the three studies differing over the size of the 
advantage of private schools (Witte, 2000, chapter 6; Rouse, 1998; Greene et al, 1999). In terms 
of who participated in the prior Milwaukee voucher program, there was agreement that the 
private schools did not “cream skim” the best students, although choice parents, while very poor 
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and often heading single-parent households, were more educated, had higher levels of prior 
parental involvement, and placed somewhat more importance on education than otherwise 
comparable MPS parents (Witte 2000, chapter 4). Finally, researchers agreed that choice parents 
were less satisfied with their prior public schools and more satisfied with their subsequent 
(private) schools than MPS parents. 

Studies of other voucher programs have found similar results on parental satisfaction but 
have mixed, and contested results on student achievement. In Cleveland there appeared to be no 
overall differences on achievement test scores between voucher recipients and public school 
students (Metcalf et al, 2003). The same was true in Washington, DC based on the first year 
outcome report of an experimental evaluation (Wolf et al, 2007). In Florida there are a series of 
different programs including a vouchers-for-failing-schools program (recently ruled in violation 
of the Florida constitution); a program for students with disabilities; and a program awarding 
scholarships to poor students to attend private schools financed by corporate contributions in lieu 
of some state taxes. Research on these programs has thus far focused primarily on the effects of 
voucher competition on student achievement in public schools. Three different research teams 
have all reported that the Florida voucher programs have resulted in public (i.e. non-voucher) 
students achieving at higher levels, but disagree regarding how much of the public school gains 
are due to voucher competition and how much are simply due to the desire to avoid the stigma of 
failure (Figlio and Rouse 2006; West and Peterson 2005; Greene and Winters 2003). 

Research Questions in the Longitudinal Educational Growth Study 

A number of important research questions surrounding educational choice reside within 
the domain of “participant” effects that is the focus of the LEGS. They fall into two broad 
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categories. The first concerns the effects of the Choice program on students themselves. The 
second is an understanding of how parents and students react to choice environments and how 
these families are impacted by such programs. The research in this study covers both of these 
areas in which school choice might affect participants. 

Program Impacts on Students. There are two general sets of issues that have repeatedly 
been analyzed in choice intervention studies, as well as studies comparing public and private 
schools. The two issues deal with educational achievement and attainment. The former, relying 
on standardized tests as key outcome variables, is often the major focus of education evaluations. 
To the extent that standardized tests are proxies for meaningful educational outcomes, these 
studies are necessary and valuable. Achievement gains are also important in that they are linked 
to what we know absolutely has lifelong value - educational attainment. Attainment is usually 
measured by years of education completed or by reaching certain milestones, such as high school 
graduation. Hundreds of studies have linked the number of years of education completed to 
outcomes such as higher lifetime incomes, and lower chances of living in poverty, incarceration, 
or having children out of wedlock. Thus both achievement and attainment are crucial outcome 
measures in studying the effects of educational policies and interventions. 

One of the primary research tasks for the LEGS is to compare the pupils in the MPCP to 
a similar group of students in MPS in terms of growth in both achievement and attainment. The 
LEGS Achievement study will be based on the progress of students beginning in grades 3 through 
8 on the Wisconsin Knowledge and Concepts Examinations (WKCE). The LEGS Attainment 
study will be based on following 9 th graders in 2006-07 through high school and beyond. 
Achievement outcomes will be judged by estimating growth in scale scores on the WKCE 
controlling for prior levels of achievement and other student and family characteristics. In the 
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future, we will look at attainment on a yearly basis in terms of remaining in school and 
progressing from one grade to the next. We will follow the 2006-07 9 th graders through what 
would typically constitute the four years of their high school education, and measure outcomes 
such as college enrollment or employment after either completing or dropping out of high 
school. We will also study how the LEGS components are related. For example, do 
achievement gains predict high school graduation? What are the effects of grade retention on 
subsequent achievement gains or on the probability of graduating? 

Another major issue that affects a number of aspects of education, but specifically 
achievement and attainment, is student mobility. Mobility is defined as students moving 
between schools when they have not reached a terminal grade in a school. Mobility may be 
between schools in the same system or between school systems or districts. In the last several 
decades educators and researchers have become keenly aware of the extremely high mobility 
rates for inner-city students. In the 1990s in Milwaukee such mobility at the elementary school 
level was judged to be between 22 and 28 percent per year (Witte, 2000, pp. 143-44). Thus we 
will be measuring and studying student mobility both as an outcome measure and as a variable 
that we hypothesize will negatively affect both achievement and attainment. 

Data for both these studies will come from administrative records, test records, and 
parental and student surveys of comparable groups. The precise methodology and data 
collection protocols are described below and in the appendices. 

Families and Educational Choice. Another important set of research questions revolves 
around families and choice environments. There are two broad sets of issues involving families. 
The first is to try to understand how families make school choices and what impact those choices 
may have on school enrollments. One critical issue is the characteristics of families who elect to 
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use vouchers to attend private schools. One hypothesis is that families with higher socio- 
economic status (SES) are more likely to use vouchers, thus leaving the traditional public 
schools with lower SES families and more difficult to educate children. An alternative 
hypothesis is that choosing parents will do so because their children are not doing well in 
traditional public schools and they look to the private schools as a way to improve the 
educational outcomes of their low-performing children. 

As noted above, earlier studies of the MPCP provided some evidence for each hypothesis 
(Witte 2000). Voucher students averaged beginning test scores that were equal to or lower than 
comparable MPS students. The Choice families also had considerably lower income and parents 
were very unhappy with their prior public schools. However, Choice parents also were more 
educated, spent more time on their children’s education, and valued education more than 
comparable MPS parents. The research in this study may shed further light on the critical 
question of whether voucher programs primarily serve advantaged or disadvantaged students. 

Family characteristics are also fundamental to understanding achievement and attainment 
issues. Parent education, parent involvement, and family status all have been linked to higher 
student achievement and attainment. Thus it is imperative to have controls for these variables in 
estimating the effects that private or MPS schools may have on achievement and attainment 
gains. 

A second set of family related issues, affecting both parents and students, are the levels 
and changes in attitudes toward the schools students attend. How do they judge their schools 
overall? How satisfied are they? How do they “grade” their schools? More specifically, what 
aspects of schools do they find most and least effective? The answers to these questions will 
provide us with clues as to what parents believe to be most important in schools, and if programs 
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like the choice program allow more families to find schools with the qualities they desire. Given 
that higher income families have more residential and hence public school choice, this study will 
seek to determine if the Choice program facilitates parental matching of schools for lower 
income families. The answer to that question may come from parent surveys over time, but also 
from student mobility between schools and into and out of the program. 

METHODOLOGY 

One of our first research tasks was to identify two comparable samples of students — one 
from the MPCP and one from MPS. Samples were required (rather than entire populations) 
because of resource limitations on surveying parents, testing students, and tracking students over 
time. After estimating the size of the samples needed for identifying any statistically significant 
results after five years (given anticipated attrition), a grade-stratified random sample of MPCP 
students was drawn for grades 3 to 8. Those grades were selected because MPS administers the 
Wisconsin Knowledge and Concepts Examinations (WKCE) annually in those grades for federal 
accountability purposes. We selected all the Choice students in the 9 th grade to follow so that we 
had a large enough sample to withstand attrition for our five-year attainment study. 4 

The question then was what to use for a comparable sample of MPS students. The ideal 
arrangement would be to have an experimental design in which all students desiring a voucher 
would be randomly selected to receive one or not. However, there is no statutory provision for 
random assignment under Wisconsin Act 125. Furthermore, since the student limits on the 



4 See Appendix C for more information on the MPCP sample selection process. 
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program had been increased in anticipation of increasing demand, it was impossible to construct 
a comparable group from waiting lists or students not picked in school and grade-level lotteries. 5 

Because the study will rely on changes over time in measures of achievement and 
attainment, it was important to be able to match samples on three critical criteria: 1) current 
levels of achievement, indicated by baseline test scores; 2) observable demographic 
characteristics known by researchers to be related to educational outcomes, and 3) variables that 
might serve as proxies for unobservable factors that may affect the comparable outcomes for 
each group. 6 To meet these criteria, we developed a unique and innovative approach in the 
construction of a sample of MPS students comparable to the representative MPCP sample. We 
used a combination of matching on specific student characteristics and a method of matching 
based on the propensity of certain students to participate in the program. Sample selection thus 
involved a three-stage process. 

In the first stage we matched students in the same grade on their neighborhoods in 
Milwaukee. We did this in sequence for each student in the sample of MPCP students. 

Following the advice of demographers and city planners, we used census tracts to identify 
student neighborhoods. Census tracts are drawn by the U.S. Census Bureau to follow 
neighborhood boundaries. In our sample, MPCP students come from 213 different census tracts. 
In this stage, for any given MPCP student in our sample, we restricted the list of potential MPS 
matches to students in the same tract. One indication that our categorization of neighborhoods 

5 Previous research on educational choice has used both of these methods. For the potential problems with using 
“rejects” from school lotteries as a control group, see Witte, 2000, pp. 136-42. However, the problem with waiting 
lists may be even more severe in that the schools with waiting lists are not randomly assigned across schools. One 
would have to assume that waiting lists indicate more desirable schools, and thus the full impact of the program (i.e. 
the full set of private schools) would not be adequately assessed using wait-listed students as a control group. 

6 Random assignment with large enough samples works to solve both of these problems. Test scores should be on 
average equally distributed between the treatment and control groups. Factors that we cannot measure that might 
lead some families to choose to participate, but could also be correlated with outcomes, are also equally distributed 
through the randomization process, thereby preventing “selection bias.” 
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has substantive support comes from the survey results outlined in Appendix D (question 58). 
These results indicate nearly identical responses from MPCP and MPS parents regarding the 
safety of their neighborhoods. In particular, roughly 8 percent of respondents from both groups 
designated their neighborhoods “very unsafe,” while 20 percent responded to the “very safe” 
designation. 

In the second stage, we matched students within census tracts within the same 5 
percentile bandwidth of WKCE baseline test scores. In the final stage, if more than one MPS 
student was matched to the MPCP student based on census tract and test scores, or if there was 
missing values for either variable for an MPCP student, we matched by estimating propensity 
scores (Rosenbaum and Rubin 1983). The propensity score matching process identifies 
characteristics commonly associated with enrollment in the MPCP that are present in MPS 
students, making those public school students a good “match” for Choice students. In this step, 
we estimated the propensity of MPCP participation as a function of the mean of math and 
reading baseline test scores, gender, race and an indicator for students with English Language 
Learning status. The MPS student with the closest propensity score to the MPCP student was 
then selected. If missing predictors made it impossible to construct a propensity score for the 
MPCP student, a MPS student was selected at random from those remaining after matching on 
census tract and prior test. If the missing predictor was student test score, matches were made at 

' 7 

random within tract. 

Propensity scores have been increasingly used in recent evaluation research when random 
assignment is not possible. We believe that by first selecting students using neighborhood and 
prior test scores and then using propensity scores to break ties or augment matches, we improve 
on simply matching on or controlling for baseline tests scores and therefore reduce the threat of 
7 See Appendix B for a detailed discussion of the sample construction. 
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selection bias. The leverage on selection effects hinges on the assumption that similar 

Q 

neighborhoods will serve as a control for unobserved variables that may affect outcomes. 

Although our future models for estimating gains in achievement and attainment will also 
include a large number of observed control variables that we know affect achievement, it is 
important to note that our matching method results in a comparison sample of MPS students with 
very similar observable characteristics to the MPCP sample. As indicated below in Table 1 and 
Figure 1 our matched MPS sample is closer to the MPCP sample on test scores, gender, and race 
than to an obvious alternative comparison group: a random sample of MPS students. 9 

Figure 1 indicates that for test scores for students in grades 3, 4 and 5, the MPS students 
matched via our algorithm (MPS Matched) are considerably closer to our MPCP sample than 
scores from a random sample of MPS students. Test scores for the various groups converge in 
the latter grades. If the goal for selecting a comparison group is to simply align baseline test 
scores as closely as possible to ensure that any diverging scores in future years are compared to a 
similar starting point, our method is at least as accurate as other methods in grades 6, 7 and 8, 
and preferable to other methods for the earlier grades. The fact that, in our comparison group, 



8 Evidence for neighborhood effects on social outcomes is presented across several social science disciplines. See, 
e.g., Aaronson (1998) for evidence of neighborhood effects on educational outcomes even after family 
characteristics are taken into account; Ludwig, Ladd and Duncan (2001) and Leventhal and Brooks-Gunn (2004) for 
experimental evidence linking neighborhood improvements to improvements in student outcomes; and Sampson, 
Morenoff and Gannon-Rowley (2002) for a general discussion. See also Cullen, Jacob and Leavitt (2005) for use of 
census tract information in research on school choice. 

9 After creating the random sample of MPS students, we compared this sample to the population of MPS students. 
We found some significant differences between our random sample and the MPS population. It is unclear if a closer 
match would be to use a random sample of all MPS students or just those MPS students eligible for reduced free 
lunch. The MPCP income limit is 175% of the poverty line or less; the reduced free lunch cutoff is 185% of the 
poverty line. Eighty-three percent of MPS now qualify for free lunch, with a number undoubtedly under identified 
(especially at the high school level). Complicating the decision is that some students in MPCP can now have 
incomes up to 220% of the poverty line, but no current data exist on how many. The issue is not critical because our 
major comparison will be with the matched MPS sample we constructed using the procedures outlined above. For 
this report we merely included a random sample of all MPS students as another possible comparison. 
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students’ neighborhood, race, gender, ELL status and the joint effect of these characteristics on 
school choice (students’ propensity scores) are accounted for further supports our method. 



Figure 1 : Combined Math/Reading Comparisons (2006-2007) 



3 4 5 6 7 8 

Grade 

Table 1 displays demographic data on students in our MPCP sample and students in both 
of the potential comparison groups. Perhaps the most important statistic is the proportion of 
white students in each sample, which indicates disproportionately high participation in MPCP 
among minority students. Nine percent of students in the MPCP sample and our MPS Match 
sample are white compared to 15 percent in the MPS Random sample. In both the MPCP and 
MPS Matched samples, African-Americans represent roughly two-thirds of the students, 
compared to 60 percent in the Random MPS sample. Compared to the MPCP sample, Hispanics 
are slightly underrepresented in the MPS Matched sample, where the 20 percent of students 



-♦-MPCP 
— ■— MPS Matched 
MPS Random 
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classified as Hispanic is closer to the MPS Random sample. The MPCP sample and our MPS 
Matched sample are more similar in gender and ELL breakdowns than the MPCP sample is to 
the MPS Random sample. 



Table 1: Comparisons of Student Characteristics in Three Possible Study Samples, 2006-07 



Sample 


Black 


Hispanic 


White 


Female 


ELL 


FRL 


MPCP 


0.66 


0.23 


0.09 


0.55 


0.11 


0.87 


MPS-Matched 


0.67 


0.20 


0.09 


0.53 


0.10 


0.86 


MPS-Random 


0.60 


0.20 


0.15 


0.48 


0.13 


0.84 



Note: Cell values represent proportions of the total sample. 
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BASELINE DESCRIPTIVES 

A Snapshot in Time: Baseline Test Scores in MPS and MPCP 

The matching procedure was necessary to produce appropriate samples for comparisons 
between MPS and MPCP students in subsequent years of the study. Because the matching 
procedure included neighborhood, student characteristics, and baseline test scores, future reports 
will be able to estimate the effects of MPCP participation on student educational growth. These 
“value-added” estimates for individual students are now the standard in estimating the effects of 
education interventions. What we will be estimating in the future are gains in educational 
achievement for individual students, comparing the results for the MPCP Random sample and 
MPS Matched sample. In this report, however, only a general description of the baseline test 
scores of the MPCP sample, the MPS Matched sample and the MPS Random sample of students 
are possible. This description is simply a “snapshot” in time and cannot be used as evidence of 
any effect of MPCP participation on student achievement. 

Table 2 reports average math and reading scale scores on the Wisconsin Knowledge and 
Concepts Examinations (WKCE) for 3 rd through 8 th graders among MPCP, MPS Matched 
students and a grade-stratified random sample of MPS students. 10 We include the random MPS 
group for additional comparison and to further indicate the success of our matching procedure. 
The table also reports the combined scale score — the mean of math and reading scores — for the 
three groups. Results from difference-in-means tests between MPCP and the two MPS groups 
are indicated in the table key. 



1(1 For more information on testing protocols see Appendix C. 
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Table 2: Sample WKCE Scores (Grades 3-8), 2006-07 



Grade 


Sample 


N 


Mean 

Reading 

Scale 

Score 


Reading 

SD 


Mean 

Math 

Scale 

Score 


Math SD 


Combined 

Mean 

Scale 


Combined 

SD 


3 


MPCP 


341 


427.9 


44.1 


380.5 


48.1 


404.1 


42.6 


3 


MPS-Matched 


341 


429.9 


42.9 


390.1** 


47.8 


410.1* 


41.1 


3 


MPS-Random 


341 


440.3*** 


38.2 


401.8*** 


49.8 


421 7*** 


40.3 


4 


MPCP 


324 


436.0 


49.4 


414.8 


49.8 


425.3 


44.7 


4 


MPS-Matched 


324 


437.5 


49.8 


423.0** 


49.9 


430.0 


45.2 


4 


MPS-Random 


324 


447.2*** 


53.3 


434 . 4 *** 


46.6 


440.8*** 


46.5 


5 


MPCP 


338 


441.8 


47.2 


437.9 


43.2 


439.6 


41.9 


5 


MPS-Matched 


338 


440.6 


51.1 


444.6** 


41.5 


442.7 


41.0 


5 


MPS-Random 


338 


448.2 


56.4 


452.4*** 


44.3 


450.8 


46.7 


6 


MPCP 


330 


463.7 


48.4 


467.5 


38.5 


465.4 


39.8 


6 


MPS-Matched 


330 


466.3 


50.0 


471.5 


42.5 


468.9 


41.4 


6 


MPS-Random 


330 


464.3 


54.3 


469.4 


43.7 


467.9 


43.8 


7 


MPCP 


303 


472.0 


51.0 


492.1 


44.9 


481.3 


46.6 


7 


MPS-Matched 


303 


467.9 


49.8 


494.2 


41.9 


481.1 


42.5 


7 


MPS-Random 


303 


476.7 


48.9 


498.6* 


46.5 


488.1* 


44.0 


8 


MPCP 


290 


487.2 


53.9 


495.9 


42.9 


490.7 


44.0 


8 


MPS-Matched 


290 


483.6 


58.8 


500.3 


46.6 


492.2 


47.6 


8 


MPS-Random 


290 


488.4 


55.3 


497.3 


50.1 


493.0 


49.5 


TOTAL 


MPCP 


1926 


453.5 


53.1 


445.7 


61.1 


449.4 


53.0 


TOTAL 


MPS-Matched 


1926 


453.5 


53.9 


452 1 *** 


59.6 


453.0** 


51.8 


TOTAL 


MPS-Random 


1926 


460.0*** 


54.0 


457 2*** 


58.0 


459.2*** 


51.6 



Note: “SD” stands for standard deviation. 



***Different from MPCP at p<0.01 
**Different from MPCP at p<0.05 
*Different from MPCP at p<0.10 



As expected by our algorithm, the MPCP students are generally very close to the MPS 
matched sample on most tests. And where there is a statistically different mean, the mean 
differences are quite small. Thus, as outlined in the last section, our algorithm for constructing 
the MPS Matched sample worked very well. 

When compared to the MPS random sample, the MPCP Matched test scores are generally 
closer to the MPCP scores. When there are statistically significant differences between MPCP 
and MPS Matched test scores, the MPS Matched scores are still closer to the MPCP scores than 
the MPS random sample scores are. Difference of means tests indicate that test scores in reading 
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for students in grades 3 to 5 were generally higher in 2006 for MPS students than for MPCP 
students, but that these differences disappear for later grades. The trend in math is slightly more 
persistent, as MPS students also score higher in grade 7. Slight but significant differences 
between MPCP and MPS Matched students are evident in math achievement for grades 3 to 5. 
However, with one exception (grade 3), these differences are not found in the combined score. 

The previously discussed Figure 1, and Figures 2 and 3 below, provide pictorial 
summaries of the results reported in Table 2. We note three patterns. The first is that scale scores 
in each grade and each subject increase by grade. The second feature is the apparent gap between 
MPCP and MPS students in early years in math achievement, and the relative lack of such a 
difference in reading. That is consistent with the data in Table 2. The final pattern is the 
convergence of math and reading scores between each group as grade increases. Although a 
slight difference in reading between MPCP and MPS students is apparent in grade 3, and a 
pronounced difference in math in grades 3-5, these differences nearly disappear in grades 6, 7 
and 8. Thus whatever baseline differences there are between these groups they are due to lower 
math scores in the early grades for the MPCP students. 
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Ninth graders in MPS do not take the WKCE. Instead they are administered the 
Benchmark Exam, which we similarly administered to the 801 MPCP 9 th graders in our sample. 
Because we matched test scores by grade, this did not require a separate matching procedure, 
although the test scales were different (scale scores for the WKCE; percent proficient in subject 
for the Benchmark). 11 We report the results of 9 th grade Benchmark scores in Table 3. As in 
earlier grades, the MPCP 9 th graders are closer to the MPS Match than to the random sample of 
MPS students in the same grade. Unlike the earlier grades, however, MPCP 9 th graders appear to 



11 The Benchmark test is not nationally normed and is used primarily as a classroom diagnostic test given multiple 
times in a year. We used it for matching purposes because it was all that was available as a measure of baseline 
achievement for the 9 th graders in each sample. We will not include these ninth graders in the LEGS Achievement 
Study. Whether we include these test scores as covariates in the LEGS Attainment Study will be determined next year 
when we will have 10 th grade WKCE scores for the 9 th graders still in school in 2007-08. 
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exceed their MPS counterparts in one subject area (reading), and this difference carries over to 
the combined math/reading scores. Observed with the pattern of converging scores (albeit from 
separate testing instruments) evident in Figures 1 to 3, these results indicate relative parity 
between MPCP students, the MPS Match students, and the MPS population in the later grades. 
Across the grade levels, there is little evidence that MPCP schools are “cream-skimming” the 
“best” students, at least in terms of test scores. 



Table 3: Sample Benchmark Scores (Grade 9), 2006-07 



Grade 


Sample 


N 


Mean 

Reading 


Reading 

SD 


Mean 

Math 


Math SD 


Combined 


Combined 

SD 


9 


MPCP 


801 


49.6 


18.8 


42.6 


16.7 


46.1 


15.6 


9 


MPS-Matched 


801 


46.2*** 


19.1 


43.9 


18.2 


44.8 


16.3 


9 


MPS-Random 


801 


445 *** 


20.6 


42.8 


18.9 


42 7*** 


18.2 



*** Different from MPCP at p<0.01 
**Different from MPCP at p<0.05 
*Different from MPCP at p<0.10 



Parent and Student Views: MPCP and MPS Survey Results 

Introduction 

In order to more fully understand the school choice environment in Milwaukee, we 
surveyed the parents and students of our MPCP sample and the MPS Matched sample. Westat 
telephoned MPCP and MPS Matched parents. MPCP students in grades 4 to 9 and their MPS 
matches also completed surveys subject to parental consent. During the testing period, MPCP 
students were given a written survey to be completed on their answer key. Those students who 
did not complete the survey at this time were noted and were later telephoned, as were all MPS 
Matched students. 12 



12 For more information on the survey protocols, see Appendix C. 
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The response rates for the parent and student surveys are presented in Table 4. For the 
MPCP parent survey, two different samples are offered. The first MPCP column includes 
response rates for the parents of students in the final sample (n = 2727) plus those parents who 
refused to have their children participate in the testing portion of the study (n = 134). Although 
we did not analyze the test scores for any student whose parent refused participation in the study, 
we did include these parents and their children in the survey sample. The total completed survey 
response rate for this sample was 64.9 with 12.3 percent of the sample completing surveys in 
Spanish. This rate is quite impressive given the mobility of the survey population. The second 
MPCP sample pertains to the response rates for the final sample only (n = 2727). Excluding 
study refusals the MPCP response rate was 65.4 percent. The response rate for MPS parents was 
51.6 percent. The MPS survey was conducted later in the year, so it is unsurprising that the 
number of parents who were not locatable was higher (39.3) in the MPS sample than the MPCP 
sample. It is encouraging that only 1.8 percent of MPCP parents and 3.9 percent of MPS parents 
outright refused to be interviewed once they were contacted by survey administrators. 

The response rates for the student survey are in the bottom of Table 4. Since third grade 

students were not surveyed, the sample sizes are smaller for the student survey than the parental 

survey. MPCP students could complete the survey in one of two ways: with paper and pencil at 

the time of test taking, or in a phone survey. Almost three-quarters of MPCP students responded 

on the paper survey. The total completed survey response rate is 83 percent for the total sample. 

All of the MPS student surveys were completed over the phone. The MPS student response rate 

is 46.6 with 4.0 percent of MPS student surveys completed in Spanish. 

Table 4: Response Rates 
2006-2007 



Parental Survey 



MPCP 



MPCP 



MPS 
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Total Sample With 
Testing Parental 
Refusals 


Final 

Sample 


Matched 

Sample 


Completed-Total 


64.9 


65.4 


51.6 


Completed-English 


52.5 


52.6 


47.0 


Completed-Spanish 


12.3 


12.8 


4.6 


Partial Completion 


0.1 


0.2 


1.1 


Ineligible 


0.8 


0.8 


0.4 


Language Problem 


0.2 


0.2 


0.4 


Refused to be Interviewed 


2.0 


1.8 


3.9 


Not Locatable 


23.0 


22.5 


39.3 


Missing 


9.0 


9.1 


3.2 


TOTAL (N) 


2,861 


2,727 


2,727 



Student Survey 


MPCP 


MPCP 


MPS 




Total Sample With 
Testing Parental 
Refusals 


Final 

Sample 


Matched 

Sample 


Completed-Total 


83.0 


84.5 


46.6 


Completed-Paper Survey 


74.6 


77.8 




Completed-Phone Survey 


8.4 


6.8 


46. 6 A 


Child Disabled 


0.1 


0.1 


0.5 


Child Moved 


0.7 


0.7 


2.2 


Ineligible 


0.6 


0.7 


0.6 


Language Problem 


0.0 


0.0 


0.5 


Refused to be Interviewed 


0.3 


0.3 


4.7 


Not Locatable 


8.0 


6.6 


41.2 


Missing 


7.2 


7.1 


3.7 


TOTAL (N)* 


2,499 


2,386 


2,386 



Note: The cell values represent percentages. 

*There are fewer student surveys than parental surveys because 3rd graders were not 
surveyed. 

A 4.0 percent of the MPS student surveys were completed in Spanish. 

The survey results provide us with a nuanced understanding of education circumstances. 
The main purposes of the survey were: (1) to understand who participates in the program; (2) to 
understand the school choice environment in the city of Milwaukee; (3) to understand parental 
participation in and expectations for their children’s achievement; and (4) to gauge levels of 
parental satisfaction with their children’s educational experiences. In the future, we will be able 
to use information from the surveys to model growth in student achievement. 



Who Participates in the Choice Program ? 
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The complete results of the parental survey are available in Appendix D. The MPCP 
parental results include 1,856 fully completed surveys and 4 partially completed surveys from 
those in the total sample (including testing parental refusals). Results from 1,438 completed 
surveys from MPS parents are also included. Analyzing the demographic characteristics of 
parents provides some insight into who chooses and who does not. However, one must remember 
that we matched students based on test scores, census tracts and, in some cases, student 
demographic information. Therefore, we did not expect large differences between MPCP and 
MPS parents, at least not at baseline. 

Race. The majority of survey respondents are African-American (Q37). While a greater 
percentage of MPS (62.5 percent) respondents were African-American when compared to the 
MPCP (57.6 percent), Hispanics make up a greater proportion of MPCP respondents (24.5 
percent) than in MPS (18.7). The number of white parent respondents is similar in each program. 



What is your ethnicity? 






MPCP 


MPS 


American Indian 


3.3 


2.2 


Asian or Pacific Islander 


2.1 


2.8 


Black 


57.6 


62.5 


Hispanic 


24.5 


18.7 


White 


15.8 


14.3 


Other/Refused/Don't Know 


1.5 


3.0 



Note: respondents could select more than one category 



Education. There are some important differences when examining the highest education 
level of respondents (Q38). About 51 percent of MPCP respondents have a high school diploma 
or less, while 14.0 percent have at least a four- year college degree. On the other hand, MPS 
respondents are comparatively less-well educated with roughly 57 percent of parents having a 
high school diploma or less and 11.1 percent having a college degree or more. This result is 
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consistent with Witte’s (2000) previous analysis of the Milwaukee choice program from 1990 to 
1994. Witte found that 45 percent of mothers with a child enrolled in the MPCP were a high 
school graduate or less, while only 8 percent had a four- year college diploma or post-graduate 
work. One of the major differences in Witte’s (2000) results and ours is that Witte found that 46 
percent of MPCP mothers had some college education, while only 30 percent of MPCP parents 
and 26.3 percent of MPS parents in our sample are in the “some college education” category. 
Thus, although MPCP parents are somewhat more educated than MPS parents, both groups are 
less well-educated now than in the 1990-94 period. 



What is the highest educational level that you have completed? 





MPCP 


MPS 


Eighth grade or below 


10.0 


5.6 


Some high school 


12.6 


17.4 


GED 


3.0 


3.8 


High school graduate 


25.2 


30.3 


Post graduate (technical school) 


4.0 


2.8 


Some college 


30.0 


26.3 


4 year college degree 


10.8 


7.8 


Post-graduate work 


3.2 


3.3 


Other/Refused/Don't Know 


1.2 


2.7 



Employment and Income. Although there are differences in the education levels of MPCP 
and MPS respondents, an examination of the employment questions finds no large differences 
(Q39 and Q40). When asked about the government assistance received by anyone in the 
household, MPCP and MPS respondents tended to reply similarly (Q59). However, MPS 
respondents were slightly more likely to receive assistance from various government programs. 

Given that there is an income cap in the voucher program, it is not surprising that MPS 
respondents have a higher income, on average (Q43). There is very little difference between 
voucher and public school parents at very low levels of income, but, at the other end of the 
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distribution, about 15 percent of MPS respondents have a total household income of more than 
$50,000, while only 4.7 percent of MPCP parents had an income in that highest category. Using 
these figures, we estimate the average income of MPCP parents to be $23,371 and the average 
income of MPS parents to be $27,577. 13 



Including everyone in your household, what was the 
total income for your household in the last calendar year 
(before taxes and other deductions)? 





MPCP 


MPS 


Less than $5,000 


8.5 


9.9 


Between $5,001 and $10,000 


12.3 


12.5 


Between $10,001 and $20,000 


23.9 


18.0 


Between $20,001 and $35,000 


31.1 


23.7 


Between $35,001 and $50,000 


13.0 


12.9 


$50,001 or more 


4.7 


15.0 


Other/Refused/Don't Know 


6.5 


8.0 



Home Lives. The survey results allow us some insight into the home lives of the MPCP 
and MPS students. We find that MPCP students are about as likely as MPS students to be living 
with married parents (38.2 percent vs. 35.7 percent). This is an increase for both groups from 
1990-1994 (23 percent for MPCP, 35 percent for low-income MPS families). According to our 
survey, about 57 percent of MPCP and MPS respondents said that the child’s other parent or 
guardian does not live in the household (Q42). There is almost no difference in family size for 
the two groups, while MPCP respondents are slightly more likely to own their own home (41 
percent vs. 38 percent) (Q56). Given that there are more Hispanics in the MPCP sample than the 
MPS sample, it is unsurprising that more MPCP respondents said that they speak Spanish at 
home with their children (19.3 percent) than MPS respondents (1 1.3 percent). 



13 Calculated by taking averages of the response categories: for the middle four categories we simply took the 
middle of the range (e.g. respondents in category 3 were given a value of $15,000). For the lowest income category, 
respondents were given an income value of $3,750, and for the highest category respondents were given a value of 
$62,500. 
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Home Life 2006-2007 





MPCP 


MPS 


Married 


38.2 


35.7 


Single parent household 


57.9 


57.4 


Spanish spoken in household 


19.3 


11.3 


Lived in Milwaukee more than 25 years 


50.8 


53.4 


Homeowners 


41.0 


38.0 



Finally, there are few differences between and MPS and MPCP respondents when asked 
how long they have lived at their current address (Q55). When asked how long they have lived in 
Milwaukee, the majority of both MPS and MPCP parents answered more than 25 years (Q57). 

Religion. Since 1996 religion and religious schools have played an integral role in the 
Choice program. We asked respondents several questions regarding the role of religion in their 
lives. In terms of religious preferences, there are more Catholics (30.0 percent vs. 20.5 percent) 
and Lutherans (9.1 percent vs 3.3 percent) in MPCP than MPS (Q52). Twelve percent of MPS 
parents said that they had no religious affiliation, while only 5.9 percent of MPCP parents 
answered similarly. When trying to understand why some parents participate in the choice 
program and some do not, the level of religiosity may be as important as parents’ religious 
preferences. MPCP respondents are more religious than MPS respondents (Q54). Almost two- 
thirds of the MPCP parents said that they attend religious services once a week or more, while 
only about 54 percent of MPS parents said the same. 



What is your religious preference? 





MPCP 


MPS 


7th Day Adventist 


1.1 


0.4 


Apostolic/Pentecostal 


7.2 


7.2 


Catholic 


30.0 


20.5 


Christian, Non-Denominational 


16.8 


20.1 


Church of God in Christ 


2.5 


2.2 


Islamic 


2.9 


1.0 


Jewish 


1.1 


0.1 
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Lutheran 


6.3 


3.3 


Baptist 


21.2 


24.3 


None 


5.9 


12.0 


Other/Refused/Don't Know 


5.0 


8.9 



The Choice Environment 

Tenure in Schools. Beyond demographic information, the survey results provide a deeper 
understanding of the education environment in Milwaukee. For example, there is great mobility 
in Milwaukee’s schools. About 39 percent of MPCP students and 37 percent of MPS students 
have been at their school for one year or less (Q2). On the other hand, 41.2 percent of MPCP 
students have been in their schools for three or more years. There are, however, clearly some 
highly mobile families in that 28.2 percent of MPS parents and 21.3 percent of MPCP parents 
said that their child has attended four or more schools (Q20). 



How many years has your child been at this school? 



MPCP MPS 



Less than 1 Year 


10.9 


3.7 


1 Year 


28.1 


33.1 


2 Years 


13.5 


20.7 


3 Years 


15.3 


14.5 


4 Years 


9.8 


7.8 


More than 5 Y ears 


22.1 


20.1 


Other/Refused/Don't Know 


0.3 


0.1 



Choosing Schools . The many and varied school choice options for parents to consider 
make Milwaukee an interesting case study. Less than one half (45.1 percent) of MPS students 
attend their residentially assigned school (Q4). Before choosing a school, MPCP parents visited 
or contacted about one public school and one private school, whereas MPS parents contacted or 
visited 1.6 public schools but only .3 private schools, on average (Q18). 
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Is this your neighborhood assigned 
or residentially assigned school? 





MPS 


Yes 


45.1 


No 


52.8 


Other/Refused/Don't Know 


2.1 



Parental information levels are an important component of school choice programs. 

About 38 percent of MPCP parents learned about the voucher program from friends or relatives 
(Q7, first response). In the same way, over 33 percent responded that they heard of the program 
from their school. The importance of social networks is also evident in choosing a specific 
school. When asked how they initially heard of their child’s school, the first response of 54 
percent of MPCP parents and 42 percent of MPS parents was from friends or relatives (Q15). 

The major difference between the groups regarding how they heard about their current schools is 
that MPCP parents received information from their church (9.4 percent) or from other private 
schools (4.5 percent) while MPS parents did not. 

On the MPS side, roughly 60 percent of parents had heard of the Milwaukee Parental 
Choice Program but only 13.8 percent of them had applied for a voucher at one point (Q8, Q9). 
Of those that applied, roughly 14 percent had children that previously participated in the MPCP, 
while about 15 percent were ineligible for the program (Q10). We also asked those who did not 
apply to the MPCP why they did not apply (Q1 1). The majority blamed a lack of information: 
43.3 percent said they did not know the MPCP existed, 18.6 percent said they did not know 
enough about the program, and 1 percent said they did not know enough about individual Choice 
schools. Another one-fifth of MPS respondents said that they did not apply to the MPCP because 
they were satisfied with their current school. 

How did you learn about the Parental Choice Program, 
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also known as the voucher program? 



MPCP 



Friends or Relatives 


1st Response 

38.4 


2nd Response 

24.4 


My Child's School 


33.5 


32.0 


Other Private Schools 


5.2 


10.5 


Newspaper/Magazine 


4.4 


8.1 


Church 


4.0 


4.7 


Television or Radio 


3.6 


12.2 


Community Center 


1.2 


1.7 


Internet 


0.7 


0.6 


Other 


8.5 


5.8 


Refused/Don't Know 


0.5 


0.0 



How did you initially hear about your child’s current school? 

MPCP MPS 


Friends or Relatives 


54.0 


41.6 


Church 


9.4 


0.3 


Other Private Schools 


4.5 


— 


Flyers/Brochures 


3.7 


4.6 


Call From School 


2.8 


3.9 


Community Center 


2.2 


2.2 


Community Events 


1.7 


0.5 


Newspaper/Magazine 


1.3 


0.9 


Home Visit 


1.0 


0.2 


Television or Radio 


0.9 


0.8 


Internet 


0.6 


0.8 


Other 


17.3 


43.7 


Refused/Don't Know 


0.6 


1.1 



Note: Parents asked to choose all that apply. First response. 



Charter schools play a prominent role in Milwaukee’s education environment. MPS 
parents (72.0 percent) were more likely than MPCP parents (65.1 percent) to have heard of these 
public schools of choice (Q12). Of those who have heard of charter schools, roughly one-fifth of 
both MPCP and MPS parents have applied to a charter school (Q13). About 7 percent of MPCP 
parents and 10 percent of MPS respondents said that their child has attended a charter school. 
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What do parents want in their children’s schools? The survey results indicate that MPCP 
parents and MPS parents have similar desires (Q16). The educational quality of the school is by 
far the most important characteristic, while student safety and teacher quality are also quite 
important (Q17). The largest difference between MPCP and MPS parents concerns the 
importance of religious instruction. About 9 percent of MPCP parents considered religious 
instruction their most important criterion, while only 1.7 percent of MPS parents did. In a school 
choice environment with many possible options, roughly three-fourths of both MPCP and MPS 
parents said that their child’s school was their first choice (Q3). 



Which school characteristic is the most important? 





MPCP 


MPS 


Educational quality of the school 


53.2 


48.5 


Teacher quality 


11.9 


16.4 


Safety in the school 


11.3 


16.6 


Religious instruction 


8.6 


1.7 


Discipline in the school 


4.3 


3.6 


School leadership 


2.7 


3.0 


Financial considerations 


1.6 


0.4 


Class size 


1.6 


2.6 


Location of the school 


1.3 


1.9 


Extracurricular activities (sports, etc.) 


1.3 


1.0 


Racial diversity 


0.9 


1.5 


School facilities (library, gym, textbooks) 


0.5 


0.6 


Special programs offered by the school 


0.3 


1.5 


Other children in the family attending the same school 


0.2 


0.2 


Other/Refused/Don't Know 


0.3 


0.5 



Was this school your first choice for your child? 





MPCP 


MPS 


Yes 


77.9 


74.1 


No 


21.7 


23.9 


Other/Refused/Don't Know 


0.4 


2.0 



Parental Involvement, Expectations, and Importance of Education 
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The survey included several measures of parental involvement. MPCP parents were more 
involved in school activities than MPS parents, including volunteering at their child's school, 
attending parent/teacher conferences, taking part in the activities of a parent/teacher 
organization, and belonging to other education organizations (Q26). In terms of contact with 
schools (Q25), results were mixed between the two groups. Both groups were contacted most 
often concerning their child’s academic performance. MPCP parents were more likely to be 
contacted for volunteering or fundraising. 



Did you (or someone in your household) do any of the following at 
your child’s school this past year? 



MPCP MPS 



Volunteer at your child’s school 

Attend parent/teacher conferences 

Take part in activities of a parent/teacher organization 

Belong to other organizations dealing with school matters 

Note: cells report percentages of respondents answering “yes.” 



54.7 
94.5 
47.4 

26.7 



37.9 

91.8 

32.1 

18.5 



We also asked parents how many times in a normal week they participate in activities 
that are educationally beneficial to their children. On this measure, MPS respondents are more 
likely than MPCP respondents to participate in these types of behaviors (Q27). About 74 percent 
of MPS parents said that they help their child with their homework three or more times a week, 
while 62 percent of MPCP parents do the same. Likewise, MPS parents were more active than 
MPCP parents in reading with their children, working on math, helping with writing and 
watching educational television programs. One might think that MPS parents are more active 
than MPCP parents because MPS students are assigned more homework. We find no evidence of 
this, as MPCP students report spending more time on homework than MPS students (Q29). 



Weekly Participation in Child’s School-Related Activities* 
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MPCP MPS 



Help your child with homework 


62.0 


73.7 


Read with or to your child 


45.9 


55.2 


Work on arithmetic or math 


46.1 


59.3 


Work on penmanship or writing 


30.8 


44.2 


Watch educational programs on TV with your child 
*Percent answering three or more times per week 


40.2 


45.7 



When asked about the educational expectations they have for their child, MPCP parents 
were more positive than their MPS counterparts. Over 54 percent of MPCP parents expect their 
child to graduate from college or go to graduate school, whereas only 46.7 percent of MPS 
parents expect the same (Q33). Likewise, 16.2 percent of MPS parents believe their child will 
achieve a high school diploma or less, as opposed to 8 percent of MPCP parents. In addition, 
MPCP parents believed their school had higher expectations for their children. Roughly 47 
percent of Choice parents strongly agreed with the statement that their child’s school has high 
expectations for academic achievement compared to 36.1 percent of MPS respondents (Q31). 

How far do you expect your child to go in school? 





MPCP 


MPS 


Finished some high school 


0.3 


1.7 


Graduated from high school 


7.6 


14.5 


Go to vocational school after high school 


1.3 


1.9 


Go to college 


35.7 


33.5 


Graduate from college 


29.7 


33.7 


Go to graduate school (law, medicine, masters degree) 


24.5 


13.0 


Other/Refused/Don't Know 


0.9 


1.7 



In order to measure how important education is to parents, we asked them to compare the 
importance of education to other goals (Q34). For example, about three-fourths of both MPCP 
and MPS parents said that education is more important than having a good job. Examining the 
survey results for these questions, one finds few differences between MPCP and MPS 
respondents. Overall, MPCP parents are slightly more likely to say that education is just as 



April 7, 2008 



36 




Education Working Paper Archive 



important as these other goals. In another attempt to determine the importance of education to 
parents, we asked them if they have ever moved so that their children may attend a better school. 
We found that this is a relatively rare occurrence, as only 15.7 percent of MPCP respondents and 
14.1 percent of MPS respondents said that they moved for this reason (Q30). 



How would you rate the importance of education in family compared to other goals? 



Education is more important Education is just as important 





MPCP 


MPS 


MPCP 


MPS 


Having a good job 


75.1 


77.0 


23.8 


21.8 


Having enough money in the family 


59.0 


61.9 


37.5 


33.8 


Maintaining religious observances/faith 


31.7 


33.5 


56.8 


52.4 


Maintaining family ethnic traditions 


35.5 


36.9 


57.6 


53.8 


Having a healthy family 


20.4 


18.4 


62.7 


61.8 


Having a good place to live 


30.2 


25.1 


63.0 


64.8 



Parental Reports of Student Success 

The survey results provide us with the opportunity to learn more about our MPCP and 
MPS students beyond what we know from administrative data. Asked what was the average 
grade their child received in school this past year, three-fourths of MPCP parents and 64 percent 
of MPS parents said their child generally received As or Bs (Q32). According to the results from 
this question, it appears as if MPCP students in our sample received slightly higher grades than 
their MPS matches (MPCP GPA: 3.0 vs. MPS GPA: 2.8). 



What is the average grade your child 
received in school this past year? 





MPCP 


MPS 


A 


30.4 


22.5 


B 


45.1 


41.8 


C 


19.3 


26.6 


D 


2.3 


4.4 


F 


0.9 


1.6 


Other/Refused/Don't Know 


2.0 


3.1 


GPA 


3.0 


2.8 
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A relatively small percentage of MPCP students (2.7 percent) and MPS students (4.2 
percent) in our sample have physical handicaps (Q46). Those parents with handicapped children 
were also asked how well their child’s school meets their child’s needs. MPS parents seemed 
more satisfied, as 51.7 percent of MPS parents said that the facilities met their child’s needs very 
well, while only 44 percent of MPCP parents responded similarly (Q47). 

While there was only a small difference in regard to physical handicaps, there is a larger 
difference between MPS and MPCP students with regards to the prevalence of learning 
disabilities (Q48). The percentage of respondents who said that their child has a learning 
disability is twice as large in the MPS sample (18.2 percent) than in the MPCP sample (8.7). 
Discussions with our Milwaukee Principal Advisory Panel indicated that MPCP school 
personnel are less likely to identify slow learners specifically as “learning disabled” than are 
MPS school personnel. It is possible that some or even all of this large difference in the reported 
rates of learning disabled students across the two groups is due to this difference in labeling 
practices and not necessarily because MPCP schools are serving fewer learning disabled 
students. There is very little difference in opinions between public and choice school parents 
regarding how well the school meets their children’s needs regarding learning disabilities (Q49). 

According to the parent survey, MPS students were more likely to miss school and have 
been suspended than MPCP students. About 29 percent of MPS respondents said that their child 
missed at least 3 days of school in the last month as opposed to only 17.4 percent of MPCP 
parents (Q50). During the past year, 33.5 percent of the children of the MPS respondents were 
suspended for disciplinary reasons (Q51). Less than 20 percent of the MPCP respondents’ 
children were suspended in that same time frame. 
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Paren tal Reports of School Satisfaction 

Using the survey results, we can examine children’s educational experiences, as well as the 
relationships between parents and schools. Perhaps one of the most important measures of school 
effectiveness is parental satisfaction. We asked parents a battery of questions regarding their 
levels of satisfaction with their child’s school across a variety of school characteristics (Q22). 
Overall, the results reveal that parents are quite satisfied in both MPCP and MPS. For all but one 
of the 15 characteristics, the majority of both MPS and MPCP parents said that they were very 
satisfied or satisfied. The one exception concerns how much students can observe religious 
traditions in schools, as MPS parents are relatively less satisfied. While both MPS and MPCP 
parents appear quite satisfied with their children’s schools, MPCP parents are relatively more 
satisfied. This discrepancy arises as MPCP parents often said they were “very satisfied” with 
school characteristics, while MPS parents were generally “satisfied.” 
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Thinking specifically about your child’s school, 
how satisfied are you with each of the following? 





Very Satisfied 


Satisfied 






MPCP 


MPS 


MPCP 


MPS 


What is taught in school 


54.9 


35.3 


36.8 


49.4 


School safety 


55.2 


36.0 


36.7 


45.8 


Amount your child has learned 


54.7 


38.3 


35.2 


41.6 


Class sizes 


49.3 


21.8 


43.8 


53.9 


Opportunities for parental involvement 


50.4 


36.8 


40.3 


47.8 


How much students can observe religious traditions 


49.0 


7.4 


40.2 


35.9 


How much teachers inform parents of student’s progress 


55.9 


41.0 


32.0 


39.4 


Academic quality 


48.8 


31.3 


41.5 


50.9 


Student engagement with school 


45.4 


28.7 


43.4 


53.4 


Teachers’ Performance 


44.7 


34.4 


43.2 


46.7 


Parental support for the school 


43.0 


27.9 


44.3 


52.0 


Principals’ Performance 


47.3 


34.8 


37.9 


42.6 


School facilities (library, gym, textbooks) 


42.5 


25.5 


44.7 


59.7 


Discipline in the school 


48.4 


30.2 


36.0 


44.4 


Transportation 


25.4 


21.8 


39.6 


50.8 



Examining the mean scores for each school attribute (Q22, Appendix D), one finds that 
MPCP parents are most satisfied with what is taught in school, school safety, and the amount 
their child has learned. This is an especially important finding because school safety and the 
educational quality of the school were the most important school characteristics for MPCP 
parents. Therefore, MPCP parents are the most satisfied with those characteristics that they deem 
to be the most important. On the other hand, MPS respondents were the most satisfied in regards 
to opportunities for parental involvement, what is taught in school and how much teachers 
inform parents of students’ progress. While both MPCP and MPS parents said that the discipline 
in school was a very important school characteristic, they were relatively less satisfied with the 
amount of discipline in their children’s schools when compared to other characteristics. 

The means indicate that the second largest discrepancy concerns class sizes. About 49 



percent of MPCP parents were very satisfied with the class sizes in their child’s school, but only 
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21.8 percent of MPS parents were very satisfied. Transportation is the one characteristic for 
which MPS parents were more satisfied that their MPCP counterparts. 

Another approach to measuring school satisfaction is to allow parents to assess problems 
that schools face in Milwaukee (Q24). Across the six dimensions we examined, MPS parents 
were more likely to say that these problems were more serious than were MPCP respondents. For 
example, 16.6 percent of MPS parents said that weapons in school were a very serious or 
somewhat serious problem in their child’s school as opposed to 6.3 percent of MPCP parents. 
Likewise, about three-quarters of MPCP parents said that fighting is not a serious problem, while 
only 53.2 percent of MPS respondents answered similarly. This finding that the parents of 
students in Choice schools view their child’s educational environment as less dangerous than do 
the parents of students in public schools has been uncovered in every previous voucher study that 
asked such questions (e.g. Wolf et al, 2007). 

In another attempt to gauge school performance, we asked parents to give a letter grade, 

A through F, to their child’s school (Q23). About 87 percent of MPCP parents gave their child’s 
school an A or B. About 75 percent of MPS respondents gave their children’s schools a similar 
high grade. Converting letter grades to grade point averages, one finds that MPCP schools (GPA 
= 3.4) received higher marks than MPS schools (GPA = 3.0). However, when comparing these 
Milwaukee results to a national Phi Delta Kappa/Gallup Poll of public school parents (Rose and 
Gallup 2007), one finds that both MPCP and MPS parents are more positive about their 
children’s schools than the national average (GPA = 2.7). 
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What overall grade would you give 
your child’s current school? 





MPCP 


MPS 


National Average 


A 


55.3 


34.0 


19.0 


B 


31.7 


40.5 


48.0 


C 


9.5 


16.9 


24.0 


D 


1.9 


4.9 


5.0 


F 


1.5 


3.1 


3.0 


Other/Refused/Don't Know 


0.1 


0.6 


1.0 


GPA 


3.4 


3.0 


2.7 



Note: National average comes from The 39th Annual Phi Delta Kappa/Gallup Poll (2007). 
Pg. 40, Table 27. The GPA calculation for the PDK poll is an approximation (n = 1,005) 



Finally, satisfaction can be measured by desires to continue in one’s current school. The 
survey results provide even more evidence of high levels of MPCP parental satisfaction in choice 
schools (Q5). Over 79 percent of MPCP parents said that they would re-enroll their child in their 
current private school next year, compared to 63.5 percent of MPS parents who gave a similar 
vote of confidence to their child’s public school. 



Do you plan to enroll your child 


in the 


same school next year? 


MPCP 


MPS 


Yes 


79.1 


63.5 


No 


20.3 


36.2 


Other/Refused/Don't Know 


0.6 


0.3 



Student Survey Results 

The complete results of the student surveys are available in Appendix E. Although 
similar, the question wording for the survey provided to students in grades four through eight 
was different than the ninth grade questionnaire. Therefore, the results we present are 
disaggregated. The results include the answers from 1415 elementary and middle school MPCP 
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students (1299 paper surveys, 116 telephone survey) and 659 MPCP 9 th grade students (560 
paper surveys, 99 telephone surveys). The MPS results for the phone survey were provided by 
795 students in grades four to eight and 318 ninth graders. 

Student’s Views of the General School Environment 

Overall, both MPS and MPCP students gave positive responses regarding the school 
environment, yet MPCP students were more positive than were MPS students. Compared to 
MPS students, MPCP elementary and middle school students had higher mean levels of 
agreement on 28 of the 35 questions, although some of the differences are quite small. Similarly, 
MPCP ninth graders were more positive about their schools than MPS students on 29 of the 35 
statements. Examining general questions regarding the school environment, the majority of 
MPCP elementary and middle school students strongly agreed that they are expected to do their 
best all the time at their school, and that they have the books and supplies they need to do well. 
These two questions also received high levels of mean agreement from comparable MPS 
students. Over 46 percent of MPCP fourth through eighth graders said they strongly agree with 
the statement that their school is a good place for learning, while only 28 percent of MPS 
students strongly agreed (Q5). 

The largest difference in mean levels of agreement for elementary and middle school 
students concerns the statement, “I have choices about what I learn” (Q20). This statement 
received the lowest mean agreement score of all 35 statements for the MPCP sample (2.7), while 
the mean score for MPS was 3.0. 
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Grades 4 to 8: Do you agree with this statement? 





Strongly Agree 


Agree 






MPCP 


MPS 


MPCP 


MPS 


At school I am expected to do my best all the time. 


61.9 


46.3 


30.9 


51.5 


The adults at my school care about the students. 


48.5 


35.7 


35.7 


56.7 


My school building is a good place for learning. 


46.7 


27.7 


40.8 


64.4 


I have choices about what 1 learn 


19.4 


19.6 


28.7 


56.5 



Among the 9 th grade general question results, both MPS and MPCP students have high 
mean agreement scores for the same two statements. The statement, “My school has high 
academic expectations of me,” received a mean score of 3.4 for MPCP students and 3.3 for MPS 
students, and the statement, “I have the books and supplies I need to do well in school,” received 
a mean score of 3.4 for MPCP 9 th graders and 3.3 for MPS. The MPCP and MPS 9 th grade 
respondents were similar in their great dissatisfaction regarding other aspects of their school 
environment. Only 8 percent of MPCP 9 th grade students and 7 percent of their MPS 
counterparts strongly agreed with the statement that students at their school focus on learning 
(Q20). Over a quarter of both groups disagreed or strongly disagreed with the statement that 
students and adults respect each other at their school (Q19). 



Grade 9: Do you agree with this statement? 





Strongly Agree 


Agree 






MPCP 


MPS 


MPCP 


MPS 


My school has high academic expectations of me. 


45.8 


35.9 


41.7 


53.1 


1 have the books and supplies I need to do well in school. 


43.4 


37.1 


49.2 


59.4 


My school building has a positive atmosphere for learning. 


33.2 


18.9 


52.5 


65.1 


I usually look forward to coming to school. 


21.7 


20.8 


49.9 


65.1 


The students and adults in my school respect each other. 


19.0 


13.5 


43.1 


54.4 


Students at my school focus on learning. 


8.4 


6.6 


51.6 


53.1 



Many of the largest mean differences between MPCP and MPS 9 th graders concern the 



general school atmosphere. MPCP students were more positive than their MPS counterparts 
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toward the statement, “My school building has a positive atmosphere for learning” (Q7), as well 
as the statement, “My school has a friendly and welcoming atmosphere” (Q10). MPS high 
school students were more positive than MPCP ones when considering two statements: “I 
usually look forward to coming to school” (Q17), and “I am pleased with my academic progress” 

(Q18). 

Safety, Discipline, and Diversity 

Beyond general questions regarding the school environment, the student survey included 
three more categories of questions: safety/discipline, diversity, and new teachers. Across these 
fifteen questions, students gave relatively positive responses. Over 60 percent of MPCP middle 
and elementary students said they strongly agreed with the statements that their school promotes 
a drug-free environment (Q21) and that adults make sure that they follow the rules at their school 
(Q22). Although similar MPS students provided mostly positive responses to these statements, 
the difference in mean agreement for Q21 (mean difference: .2) and Q22 (mean difference: .2) 
were relatively large. One of the least positive responses for MPCP and MPS fourth through 
eighth graders across these three categories was for the statement, “My school building is neat 
and clean” (Q29). 

Both MPS and MPCP students gave high marks to their schools in terms of their support 
for diversity and new teachers. Almost 80 percent of MPCP students and over 92 percent of MPS 
students strongly agreed or agreed with the statement that adults at their school help students of 
different races to get along (Q31). Forty-nine percent of MPCP middle and elementary students 
and almost 30 percent of MPS students strongly agreed that their school teaches them to value 
and respect others who are different (Q30). 
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Grades 4 to 8: Do you agree with this statement? 





Strongly Agree 


Agree 






MPCP 


MPS 


MPCP 


MPS 


My school building is neat and clean. 


31.6 


19.9 


42.0 


60.5 


The adults at my school make sure I follow the rules. 


60.3 


37.5 


32.8 


60.0 


I understand my school’s rules about behavior. 


56.2 


35.0 


35.4 


57.7 


The adults at my school help children of different races get along with each 
other. 


43.5 


34.5 


35.6 


57.9 


My school teaches me to value and respect others who are different from me 


49.1 


29.6 


39.8 


65.8 



The 9 th grade results for the safety/discipline, diversity and new teachers categories are 
similar to the elementary and middle school results, as both MPS and MPCP students gave 
generally positive responses, though the MPCP students were more positive. MPCP students had 
an almost 20 percentage point advantage (51.4 vs. 31.5) in the strongly agree category for the 
statement, “My school provides a drug-free environment” (Q21). Similarly, 42.6 percent of 
MPCP students compared to 28.9 percent of MPS students strongly agreed that their schools 
“make sure that classrooms are safe and orderly.” These results fit well with MPCP parental 
differences on satisfaction with discipline noted earlier. 



Grade 9: Do you agree with this statement? 





Strongly Agree 


Agree 






MPCP 


MPS 


MPCP 


MPS 


My school promotes a drug-free environment. 


51.4 


31.5 


37.0 


59.1 


My school makes sure that classrooms are safe and orderly. 


42.6 


28.9 


47.2 


59.1 


My school treats all students with respect regardless of their race or ethnic 


38.5 


35.5 


43.3 


52.8 



background. 
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Survey Summaries 

Our main goals and findings for the parent survey were fourfold. 

1. Who Participates in the Choice Program ? 

• MPCP parents had lower household incomes than MPS parents. 

• MPCP parents had slightly more education. 

• Family structure and sizes were very similar between MPS and MPCP families. 

• MPCP parents were significantly more religious. 

2. Understanding the Choice Environment 

• With the exception of information from churches and private schools, sources of 
information on schools were similar between MPCP and MPS parents. 

• There were also few differences in qualities parents sought in schools. 

• Over 70 percent of parents in both groups responded that they received their first choice 
of schools. 

3. Parental Involvement, Expectations, and Importance of Education 

• MPCP parents were more likely to be involved in school activities than MPS parents. 

• However, MPS parents were more likely than MPCP parents to be involved with their 
child at home. 

• MPCP parents have higher expectation in terms of student educational attainment. 

• However, there was no difference on the importance parents placed on education. 

4. Parental Satisfaction with Their Current Schools 

• Overall levels of satisfaction are very high for both groups and on “grades for their 
school” higher than the national average. 

• On a number of measures MPCP parents are somewhat more satisfied than MPS parents. 



Our main findings from the school climate survey administered to students were twofold. 

1. Overall, for both 4 th through 8 th graders and 9 th graders (studied separately), attitudes 
were very positive in both MPS and MPCP schools. 

2. In general on all the dimensions there were small differences between MPCP and MPS 
student attitudes, with these differences usually favoring MPCP schools. 



SUMMARY AND CONCLUSIONS 

This report discusses the initial stage of the Longitudinal Educational Growth Study 
(LEGS) of the Milwaukee Parental Choice Program (MPCP). The results presented here are the 
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products of a single “snapshot” in time. This report can describe the educational reality of 
MPCP and MPS parents and students, but it cannot make reliable claims regarding the extent to 
which the Choice program itself has caused any of these conditions. Only once longitudinal 
growth data have been collected can such programmatic claims be made with any confidence. 
This baseline report does, however, provide several indications of both similarities and 
differences currently existing between MPCP and public school students. Among these are: 



• WKCE math and reading scores for MPCP students in grades 3-5 are slightly 
lower at baseline than those for their MPS counterparts. These differences may be 
the result of a variety of factors unexplored in this baseline report, and the subject 
of examination in future years of the LEGS. 

• WKCE math and reading scores for MPCP students in grades 6-8 do not differ 
from the scores of the MPS students. 

• Benchmark Test results for 9 th graders are also similar between the two groups. 

• MPCP and MPS parents and students are generally satisfied with their respective 
schools. Overall, MPCP parents and students were more positive than their MPS 
counterparts. 

• MPCP parents have higher levels of education and attend religious services more 
regularly than MPS parents. MPS respondents have higher incomes, on average. 

Later reports will expand upon the results presented here at baseline. Growth in student 
achievement, the rate of student entry, exit, and potential re-entry into the MPCP program, and 
the effect of these choices on education outcomes are all subjects of future stages of the 
Longitudinal Educational Growth Study. 
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APPENDIX A 

Description of the Study Mandate 

The legal responsibility of the School Choice Demonstration Project under Act 125 is 
limited to collecting standardized data from the private schools and turning it over to the 
Legislative Audit Bureau each year beginning in 2007. The standardized test scores were 
collected from participating schools throughout the 2006-07 academic year, organized into a 
single database, and delivered to the Legislative Audit Bureau on December 28, 2007. Those 
data are discussed in the MPCP Annual School Testing Summary Report. 

This particular report is focused on fulfilling the SCDP’s vision of conducting a 
longitudinal evaluation of a representative panel of choice students closely matched to a panel of 
MPS students in order to estimate the actual effects of the MPCP on important student and 
family outcomes. This idea of a rigorous longitudinal evaluation of school choice in Milwaukee 
has been endorsed by the SCDP’s Research Advisory Board and is being supported by the six 
foundations that have thus far agreed to underwrite the School Choice Demonstration Project’s 
Milwaukee study. 

Although this project is being accomplished without state funds we are grateful to the 
state of Wisconsin for providing legislation in the form of Act 125 to authorize this important 
research project. We also thank the leadership and staff of the Wisconsin Department of Public 
Instruction and the Milwaukee Public School District for their critical assistance with this study. 
Finally, we acknowledge the extensive efforts to cooperate with this evaluation undertaken by 
the personnel at the various private schools that participate in the MPCP. We will continue to 
work closely with these entities, the legislature, the Legislative Audit Bureau, and other relevant 
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state authorities to carry out the most complete, accurate, and informative study possible within 
the confines of the data and research circumstances that we encounter. 
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APPENDIX B 

Constructing the Sample for Study 

To identify a representative sample of students to study over the 5-year duration of the 
LEGS Achievement and LEGS Attainment studies, we first selected a random sample of 
participants in the Milwaukee Parental Choice Program (MPCP) from a September 15, 2006 list 
of applied and accepted students. To obtain proportions of students for the LEGS Achievement 
study in grades 3-8 that were representative of the MPCP population, we stratified the selection 
by the number of students in each grade in the program. We then drew random samples for each 
grade, for a total of 2,184 students. For the LEGS Attainment study we selected all of those in 9 th 
grade (911 students). The samples combined for a total of 3,095 students comprising 18 percent 
of the population of all MPCP participants in grades 3-9. 

We then examined the audited list of voucher recipients on the 3 Friday count 
(September 15, 2006) from the Department of Public Instruction. 14 Two hundred twenty-seven 
students were not on this list or had duplicate records and were dropped from the study. 15 We 
informed each MPCP school as to which of their students had been selected. The parents or 
guardian of each student were informed via letter from their child’s school of their child’s 
selection, and were given the opportunity to decline participation. Of those students in the 
sample, 134 (4.7 percent) opted out of the study. An additional 7 students were not included in 
the study because their grade levels were no longer within grades 3 through 8. For those students 
who remained after these adjustments, we obtained information on students’ race, gender, and 

14 The 3 rd Friday in September is used in Wisconsin as the official enrollment count for all public schools. State aid 
and other formulas and aid programs depend on this count. 

15 The 3 rd Friday list included only students who applied, were accepted and were enrolled on that date in the private 
schools. The students who were dropped were on the original September 1 list, but were not in the schools on 
September 15 th , the third Friday of the month. That is very common in Milwaukee where students often apply to 
multiple schools under a number of choice programs (e.g. charter schools, magnet schools, and suburban schools). 
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other variables through school records. To the extent possible, for those students missing 
administrative data, we added data with student survey data administered during the baseline 
testing session. The final analytic sample was comprised of 2,727 students in the MPCP 
program. See Appendix C for additional information regarding MPCP sample selection. 

The next step in the sample construction was the selection of students in Milwaukee 
Public Schools (MPS) who would constitute valid comparisons to the MPCP participants. 
Without such a process, we would not be certain that differences in student achievement growth 
and other outcomes of interest between MPCP and MPS would be attributable to differences in 
student baseline characteristics, or differences that influence both the decision to leave or remain 
in public schools and the outcomes of interest. To obtain a valid comparison group, we designed 
a multi-step procedure to incorporate students’ neighborhood information, prior achievement 
levels, and student demographics into the selection of our MPS sample. 

The basic sample design was to first match MPCP students to MPS students on their 
neighborhood, then on baseline test scores, and then to use propensity scores for being in MPCP 
to order the MPS students already matched on neighborhood and/or baseline test. The matching 
was done without replacement, meaning that each MPS student could only be matched to one 
MPCP student. The first step in this procedure was the inclusion of the census tract 
corresponding to the home address reported for each MPCP and MPS student. Census tracts are 
geographic locations given unique identification numbers by the United States Census Bureau. 16 
In this step, potential MPS matches for a given MPCP student were limited to those in the MPCP 



16 Census tract was selected as the critical neighborhood proxy rather than a distance variable on advice of the 
Milwaukee Public Housing Authority, and after investigation that confirmed that the census tracts in Milwaukee 
were carefully drawn to represent neighborhood demarcations. 
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student’s census tract. In our MPCP sample, there were students from 213 unique tracts in 
Milwaukee. 

We next narrowed potential MPS matches within each tract for grade 3-8 students with 
similar scores on the Wisconsin Knowledge and Concepts Examinations (WKCE). Similarly, we 
matched 9 th graders based on the Benchmark Exam scores (see below). To define which scores 
were “similar” we divided the distribution of mean MPCP scale scores by grade, and then further 
into twenty test bands, each corresponding to every 5 th percentile of the grade-specific 
distribution of scores. In the first band were students whose scores fell between the 1 st and the 5 th 
percentile. In the twentieth band were students whose scores fell on or exceeded the 95 th 
percentile. We restricted test score matches for each MPCP student to those MPS students whose 
raw test scores fell within the same test band defined by the MPCP distribution. 

Finally, we narrowed potential matches further by estimating the influence of students’ 
race, gender, ELL status and baseline test score (the mean of the math and reading tests) on the 
likelihood that any student would select private education (MPCP=1). In this step, we estimated 
a propensity score, choice , using a logit model: 

choice= P(MPCP=1) = z xp / 1 + & xp Equation (1 ) 

for each student based on these characteristics, X, with models estimated separately by 
grade. Within census tracts and test bands, MPS students were matched to MPCP students by 
selecting the MPS student with the closest raw value of choice to the MPCP student. In our 
initial data collection at baseline, we were unable to gather complete data for all students in our 
sample, although the data are complete for the vast majority of students, as Table B-l indicates. 
For MPCP students whose propensity scores we were unable to estimate due to missing data on 
race, gender, or ELL we simply drew an MPS student at random from within census tract and 
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test band matches. If a missing test score caused the missing propensity score, we drew the MPS 
match at random from within the set of census tract-matched students. All selections were 
without replacement. Table B-2 summarizes the success of these matches. As the table indicates, 
nearly all (99 percent) of students were matched on census tract, while only 56 percent of 
students were subsequently matched on test band. 

After creating the matches, we verified with the 3 Kl Friday list that each MPS student was 
in the same grade as their MPCP match. For those who were not in the same grade, the matching 
procedure was repeated until a correct match was made. After the matching process was 
completed we received 211 additional MPCP test scores that were previously coded as missing, 
as well as 642 additional MPS tracts that were previously missing. We updated the tract 
information on the 642 MPS students and repeated the matching algorithm for the 211 MPCP 
students. Recently, we received an additional 158 MPCP test scores. Because of their late arrival, 
we were unable to include test scores in the matching algorithm for these students. Instead, we 
included for these students the original MPS matches made through random selection within 
census tract. As discussed in the text, Tables 1-2 and Figures 1-3 show comparisons of our 
matched students, a random sample of MPS students, and our MPCP sample. 



Table B- 



Complete MPCP Data 





Reading 

3-8 


Math 

3-8 


Reading 

9 


Math 9 


Race 


Gender 


ELL 


FRL 


N 

(% 

complete) 


1,640 

(85.2) 


1,636 

(84.9) 


583 

(72.8) 


583 

(72.8) 


2,464 

(90.4) 


2,470 

(90.6) 


2,314 

(84.9) 


2,114 

(77.5) 



Reading and math percentages for grades 3-8 based on a denominator of 1,926 students; For grade 9 scores the denominator is 801 students. 



April 7, 2008 



56 




Education Working Paper Archive 



Table B-2: Matching By Tract and Testband 



N (pet) 


No Tract Match 


Tract Match 


Total 


No Testband 


9 


1, 197 


1,206 


Match 


(0.8) 


(99.3) 


(100.0) 




(30.0) 


(44.4) 


(44.2) 


Testband Match 


21 


1,500 


1,521 




(1.4) 


(98.6) 


(100.0) 




(70.0) 


(55.6) 


(55.8) 


Total 


30 


2,697 


2,727 




(1.1) 


(98.9) 


(100.0) 




(100.0) 


(100.0) 


(100.0) 
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APPENDIX C: Data Collection Procedures and Protocols 



In this section, we describe the methodology for selecting the sample of panel students 
and the protocols used for each of the following data collection activities: 



• Administering the WKCE-CRT to panel students in grades 3-8, 

• Administering the Milwaukee Benchmark exam to 9 th grade panel students, 

• Conducting the survey of all panel students, 

• Conducting the telephone survey of parents of panel students, and 

• Conducting the telephone survey of MPS students and their parents. 

MPCP Sample Selection 

Included below is a description of the methodology used to randomly select the MPCP 
panel for the Evaluation of the Milwaukee Parental Choice Program. The design of the study 
called for a random sample of 3,000 panel students in grades 3-8 and a census of all 9 th grade 
students in schools attended by panel students. The steps taken to draw the sample are described 
below: 



• In September 2006, the Milwaukee Department of Public Instruction (DPI) provided 
Westat with an un-audited data file of 16,892 students who were expected to be enrolled 
in the Milwaukee Parental Choice Program (MPCP) during the 2006-07 school year. 

This un-audited file was used to draw the sample of panel students because the audited 
data file was not available prior to the state mandated window for administering the 
WKCE-CRT-CRT. A sample of 3,095 panel students in grades 3 through 9 was drawn. 
Please note the census of 9 th graders were included in the sample. After removing 20 
duplicate records, the panel size was reduced to 3,075 students in grades 3 through 9. 

• In late April 2007, DPI provided Westat with an audited file of 17,798 students 
confirmed to be enrolled in MPCP. Westat attempted to match the sample of 3.075 panel 
students (drawn from the un-audited file) against this audited file. We matched on the 
following variables: SCHID, Child first name, Child last name, and DOB. In order to be 
included in the baseline sample, the student’s name had to be included on the audited file. 
Using this criterion, the matching process resulted in a total of 2,869 unduplicated 
students. 
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Testing Protocol 

Prior to conducting the baseline testing, the study team prepared some informational materials to 
share with principals of participating schools to inform them about the requirements of the 
evaluation. In addition, principals received other testing related mailings in advance of the 
planned testing. The advance communication with principals of MPCP schools is outlined 
below: 



• On September 8, 2006, the co-principal investigators and members of the study team 
attended a meeting of MPCP school principals in Milwaukee to introduce the study and 
to answer questions. 

• In mid-September, 2006, Westat mailed the list of 3075 panel students (original sample 
drawn from the DPI un-audited file) to each school. As part of this process, schools were 
asked to verify the enrollment of students. At this time, schools reported that 2,894 panel 
students were enrolled. We attempted to test these students at the schools where they 
were confirmed as enrolled. 



The representative sample, or “panel,” of MPCP students in grades 3-8 were tested in reading 
and math using the Wisconsin Knowledge and Concepts Examinations - Criterion-Referenced 
Tests (WKCE-CRT). Panel students in grade 9 were tested in the same subjects using the 
Discovery Think Link Benchmark exam. These tests were chosen for panel testing because they 
were also being administered by Milwaukee Public Schools. The protocol for testing all panel 
students follows: 



WKCE-CRT Testing. Sampled panel students in grades 3-8 were administered the WKCE-CRT 
in their own schools by school personnel between November 7-24, 2006. The testing conditions 
replicated how MPS students are tested and therefore allow proper comparisons between the 
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performance of MPCP and MPS students. The testing window was required by DPI and 
coincided with testing being conducted by MPS schools. 



Benchmark Testing. The census of panel students in grade 9 were tested using the Milwaukee 
Benchmark exam between December 8-20, 2006 in accordance with DPI requirements. 



Prior to testing the panel students, the study team participated in a MPS sponsored web-cast 
training on administering the WKCE-CRT-CRT. This training formed the basis for developing 
the materials used to train MPCP staff who would be administering the WKCE-CRT to panel 
students. Listed below is a summary of the training related activities that occurred prior to 
testing. 



• Westat incorporated DPI’s specific testing protocols and requirements into training 
materials developed for the in-person training of MPCP test administrators. The training 
materials mirrored the training modules and content of the DPI training. 

• Westat conducted an on-site training of MPCP teachers and staff who would be the test 
administrators for their schools, from November 7-9, 2006. Each MPCP school was 
required to send at least one test administrator to be trained. Several important modules 
were incorporated into the training, including 

■ MPCP Study and Testing Overview 

■ Role of the Test Administrator 

■ Materials needed for testing 

■ Getting familiar with the test booklets 

■ Administering the WKCE-CRT-CRT using appropriate testing procedures 

■ Make-Up Sessions 

■ Test Security Guidelines 

■ Participation of students with special needs and accommodations 

■ Use of calculators 

• Prior to testing, schools sent home an informational letter to parents and a parental 
consent form to the parents of all MPCP panel students. Parents could choose to op-out 
of the study. Student’s whose parents opted out of the students were not tested or 
surveyed. 
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MPCP Parent and Student Surveys 

Included below is a description of the process used to survey students and their parents as part of 
the Evaluation of the Milwaukee Parental Choice Program. The design of the study called for a 
survey of the parents of all panel students, as well as panel students in grades four to nine, to 
learn about their opinions and their educational experiences. Westat’ s IRB granted approval to 
conduct the telephone survey as part of the Evaluation of the MPCP on April 3, 2007. 

Paper and Pencil Student Surveys - Fall 2006. In Fall 2006, panel students were administered a 
paper and pencil survey immediately after completing the WKCE-CRT or Benchmark exam. 

The survey was not mandatory. Students could refuse to participate in the survey component of 
the study. This survey is the same one that MPS administers every year to their students in 
grades four and higher. The question wording was slightly different in the elementary/middle 
school questionnaire and the high school questionnaire. 



Student and Parent Telephone Surveys - Spring/Summer 2007. In April 2007 Westat received 
the audited filed from DPI. Subsequently, a first attempt was made to match the 2,894 panel 
students against the audited file. 2,750 matches were found. 

• Westat sent the list of 2,750 students to the MPCP schools in an attempt to obtain up-to- 
date phone numbers for these households. (The schools could not provide updated phone 
numbers for many of these families). During this process schools informed us that 
additional students were “no longer enrolled”. 

• Flags were used in the database to identify students who were “no longer enrolled” at the 
school or who had “moved out of Milwaukee”. These students, however, remained part 
of the sample. 

• This information was used to generate a list of 2,749 MPCP students (one student was not 
enrolled at the school listed on the un-audited file) for the MPCP parent phone survey. 
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• In May 2007, Westat’s Telephone Research Center began the telephone survey of MPCP 
parents. Parents were given $20 for their participation in the approximately 30-minute 
survey. All parents were told to answer questions based on their experiences during the 
2006-2007 school year. In addition, attempts were made during the phone calls to survey 
students who had not completed the paper and pencil survey during the fall 2006 testing 
period. 

o 1,860 Parent Phone surveys were completed by the end of August, 2007. 
o 215 Student Phone surveys were completed by the end of August, 2007, bringing 
the total of Student Surveys completed (paper/ pencil and phone) to 2,074. 



MPS Parent and Student Surveys 

In July 2007, 2,727 Milwaukee Public School students were selected as the control group 

panel to the MPCP panel of students. The goal of all MPCP Evaluation data collection 

activities is to compare the results of MPCP students with those of the control group of MPS 

students, therefore the design of the study called for the survey of MPS panel students and 

their parents to learn about their opinions and their educational experiences. 

• In mid-July 2007, Westat’s Telephone Research Center began the telephone survey of the 
2,727 MPS parents and students in grades four to nine. Parents were given $20 for their 
participation in the approximately 30-minute survey. All parents were told to answer 
questions based on their experiences during the 2006-2007 school year. By mid- 
November, 

o 1,113 Student Surveys had been completed, and 
o 1,438 Parent Surveys had been completed. 



Survey Response Rates 



For the Parent Survey: 

• 64.9 percent for the MPCP sample including testing parental refusals 

• 65.4 percent for the MPCP sample excluding testing parental refusals 

• 51.6 percent for the MPS Matched sample 

For the Student Survey: 

• 83.0 percent for the MPCP sample including testing parental refusals 

• 84.5 percent for the MPCP sample excluding testing parental refusals 

• 46.6 percent for the MPS Matched sample 
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